Audiovisual adaptation is expressed in spatial and decisional codes

The brain adapts dynamically to the changing sensory statistics of its environment. Recent research has started to delineate the neural circuitries and representations that support this cross-sensory plasticity. Combining psychophysics and model-based representational fMRI and EEG we characterized how the adult human brain adapts to misaligned audiovisual signals. We show that audiovisual adaptation is associated with changes in regional BOLD-responses and fine-scale activity patterns in a widespread network from Heschl’s gyrus to dorsolateral prefrontal cortices. Audiovisual recalibration relies on distinct spatial and decisional codes that are expressed with opposite gradients and time courses across the auditory processing hierarchy. Early activity patterns in auditory cortices encode sounds in a continuous space that flexibly adapts to misaligned visual inputs. Later activity patterns in frontoparietal cortices code decisional uncertainty consistent with these spatial transformations. Our findings suggest that regions within the auditory processing hierarchy multiplex spatial and decisional codes to adapt flexibly to the changing sensory statistics in the environment.


Supplementary Note 1 Eye movement results
To control for eye-movement-related confounds, we recorded participants' eye movements during the psychophysics and EEG experiments. In the fMRI experiment, precise positioning of participants' head inside the scanner bore was critical for the sensitive measurement of spatial recalibration, which did not allow high quality eye-movement recordings. As EEG recordings are highly susceptible to eyemovement-related confounds in spatial tasks 1 , we rejected all trials in the EEG experiment that did not fulfil stringent criteria (% rejected trials across subjects mean ± SEM: 8.2 ± 1.0%; see EEG Pre-processing section for criteria).
In the psychophysics experiment fixation was well maintained throughout the entire experiment with participants fixating correctly on 94.02% ± 1.00% (mean ± SEM) of all trials (i.e., eye gaze was maintained within a radius of 2˚ and without saccades or eyeblinks) and post-stimulus saccades detected on only 1.24% ± 0.33% (mean ± SEM) of all trials (including response and non-response trials in all pre-, postAV/postVA-and AV-adaptation phases). In one-way repeated measures ANOVAs we assessed whether eye movement indices (i.e., % saccades, % eye blinks, and post-stimulus mean horizontal fixation position) differed across the 3 experimental phases (pre-, postAV-, and postVAadaptation) in the non-response trials (i.e., the trials that were included in our fMRI analysis). No significant differences were found across the 3 experimental phases for % saccades and % blinks, however, we observed a significant effect of experimental phase on the mean horizontal eye position (F(2,14) = 8.02, p = 0.002). Fisher's LSD post-hoc tests revealed significant differences only between pre-and post-adaptation phases (mean difference between pre-and postVA-adaptation = 0.061˚, p = 0.004; mean difference between pre-and postAV-adaptation = 0.047˚, p = 0.027). Importantly, we did not observe a significant difference in the mean horizontal fixation position between postAV-and postVA-adaptation phase (p = 0.147). In other words, the direction of recalibration did not affect observers' eye position. This is critical because we assess the effects of recalibration by directly comparing trials from postAV-and postVA-adaptation phases to avoid temporal confounds.

Supplementary Note 2 Assessment of decisional choice model in fMRI and EEG
To further assess whether the decisional choice model nevertheless accounts for additional variance in the regional BOLD-response or the fMRI or EEG activity patterns, we compared the winning model from the Bayesian model comparisons of the main analysis with the winning model that was extended by the 'decisional choice component'. This extended model did not explain additional variance in the regional BOLD-response (Supplementary Table 6), but it explained additional variance in the activation patterns of IPS for spatial encoding and in those of hA and IPS for spatial recalibration (Supplementary Table 7). Consistent with these results, the winning model extended with a decisional choice component explained additional variance in the EEG activity patterns from 250 to 450 ms (Supplementary Table 9). Collectively, these results corroborate the main findings reported in the main manuscript that decisional representations affect mainly later processing stages in parietal cortices. It is interesting that decisional uncertainty is more strongly reflected in IPL, while decisional choice is more strongly represented IPS and possibly via feed-back connections later in hA.

Supplementary Note 3
Pairwise correlations between the predictions of the spatial, decisional uncertainty, decisional choice, and response time models The main manuscript reports the analyses and results for the spatial, the decisional uncertainty and the combined (i.e., spatial + decisional uncertainty) models, because these models produce predictions for the regional mean BOLD-response and activity pattern similarity that are clearly distinct. We also assessed two additional computational models that were based on dependent measures such as observers' response choices and response times (see Supplementary Figure 4B). The decisional choice model codes the percentage of 'right' (i.e., positive azimuth or right-sided) responses for each spatial location x condition and participant. The response time model codes the mean response time for each spatial location x pre-and post-adaptation phases and participant. Supplementary Figure 4 shows the predictions of all four models for the regional mean BOLD-response and the neural activity patterns. Visual inspection suggests that the predictions of the decisional uncertainty model are similar to those of the response time model. Indeed, the predictions of the decisional uncertainty and the response time models were highly correlated (Spearman's rank correlation: R S (19) = 0.95, p < 0.001). This high correlation corroborates that the distance of the spatial location from observers' decisional boundary as incorporated in the decisional uncertainty model indexes decisional uncertainty. Conversely, the predictions of the spatial and the decisional choice models were highly correlated (Spearman's rank correlation: R S (19) = 0.99, p < 0.001).

Participants
Nineteen right-handed participants took part in the pre-screening session. 15 of those participants (10 females, mean age = 22.1; SD = 4.1) returned for the psychophysics part of the study. Participants were included after the pre-screening session if (i) they provided reliable spatial discrimination responses in the auditory pre-adaptation phase, i.e., the just noticeable difference (JND) of the psychometric function fitted to the spatial discrimination responses was < 4°; (ii) reliably discriminated between response and non-response trials, i.e., sensitivity index (d') > 2.5 and (iii) reliably detected the target luminance change during the audiovisual adaptation phase, i.e., d' > 2.5 (see Experiment design and procedure for details). Six of the psychophysics participants that had the highest recalibration effect size were selected to participate in the fMRI and EEG experiments (see 2 ). Five of those participants (4 females, mean age = 22.2; SD = 5.1) completed the whole study (one was an author of the study, A.M.). One participant was excluded after 3 fMRI days because their sensitivity index d' during the adaptation phase was ≤ 2.5 (see above iii). All participants had no history of neurological or psychiatric illnesses, normal or corrected-to-normal vision and normal hearing. All participants gave informed written consent to participate in the study, which was approved by the research ethics committee of the University of Birmingham (approval number: ERN_11_0470AP4) and was conducted in accordance with the principles outlined in the Declaration of Helsinki.

Auditory pre-and post-adaptation
To increase design efficiency (in the fMRI study), stimuli were presented from one of the 7 possible locations in a pseudorandomized order in ~36 s blocks of 18 stimuli interleaved with 6 s fixation. In the pre-adaptation phase one run consisted of 10 blocks (i.e., 18 stimuli x 10 blocks = 180 stimuli) and 9 fixation intervals and lasted for ~7 minutes. After five blocks we inserted a longer fixation interval of 15 s. In the post-adaptation phase, each run started with an audiovisual adaptation block (see next section) followed by 5 post-adaptation blocks. In total, a run included 2 audiovisual adaptation and 10 post-adaptation blocks (~14 minutes). As in the pre-adaptation phase, the post-adaptation blocks were interleaved with 6 s fixation. The EEG experiment followed the same overall structure as the fMRI experiment, except that the fixation intervals were not included to reduce the duration of the experiment. In the psychophysics experiment, runs in the pre-adaptation phase consisted of 5 stimulus blocks (i.e., 18 stimuli x 5 blocks = 90 stimuli) and 4 fixation intervals (~3.5 minutes). To maintain participants' engagement with the task, they received visual feedback after every 5 pre-or postadaptation blocks (i.e., 90 trials) about their detection performance (i.e., the dimming of the fixation cross, n.b. not their localization performance) based on the sensitivity index d' (i.e., d' > 3.5: smiling face, 2.5 < d' < 3.5: neutral face, 2.5 < d': sad face).
The pre-adaptation phase of the fMRI (resp. EEG) experiment included 3600 auditory trials (2800 no response + 800 response trials, i.e., 18 stimuli per block x 10 blocks per run x 5 runs per day x 4 days). The postVA (resp. postAV)-adaptation phase included 1440 auditory trials (1120 no response + 320 response trials) for postAV-adaptation and the same number of trials for postVA-adaptation phase (i.e., 18 stimuli per block x 10 blocks per run x 4 runs per day x 2 days). In the psychophysics experiment, the pre-adaptation phase included 1800 auditory trials (1400 no response + 400 response trials, i.e., 18 stimuli per block x 5 blocks per run x 5 runs per day x 4 days). The postVA (resp. postAV)-adaptation phase included 1800 auditory trials (1400 no response + 400 response trials) for postAV-adaptation and the same number of trials for postVA-adaptation phase (i.e., 18 trials per block x 10 blocks per run x 5 runs per day x 2 days)

Audiovisual adaptation
One audiovisual adaptation block included either 360 trials (for the first adaptation block in psychophysics and typically in fMRI and EEG, i.e., 3 minutes) or 120 trials (for the remaining adaptation blocks in psychophysics, i.e., 1 minute). To maintain participants' engagement with the task, they received visual feedback after every adaptation block (i.e., 360 or 120 trials) about their detection performance (i.e., of the dimmer visual stimuli) based on the sensitivity index d' (i.e., d' > 3.5: smiling face, 2.5 < d' < 3.5: neutral face, 2.5 < d': sad face). In total, both fMRI and EEG studies included 5760 audiovisual adaptation trials (5184 no response + 576 response trials) for the (left) VA-adaptation phase (i.e., 360 trials per block x 2 blocks per run x 4 runs per day x 2 days). Likewise, there were 5760 audiovisual trials for the (right) VA-adaptation phase.

Organization of each testing day
Each day started with 5 pre-adaptation runs amounting to 900 trials per day (i.e., 18 trials per block x 10 stimulation blocks per run x 5 runs). The subsequent 4 runs (resp. 5 runs in psychophysics) included 2 VA (or AV)-adaptation blocks and 10 postVA (or postAV)-adaptation blocks, such that each VA (or AV)-adaptation block was followed by 5 postVA (or postAV)-adaptation blocks. Hence, each day included 2880 trials AV adaptation trials (i.e, 360 per block x 2 blocks per run x 4 runs per day in fMRI and EEG and 2400 AV adaptation trials in psychophysics). Further, each day included 720 postadaptation trials in fMRI and EEG (i.e., 18 trials per block x 10 blocks per run x 4 runs per day) and 900 trials in psychophysics (because of 5 runs per day, see Figure 1B). The AV adaptation and postadaptation blocks were interleaved with 15 s of fixation.

Response time data analysis
Response times were recorded for the 'response trials', i.e., for 22% of trials in the pre-and postadaptation phases in which observers explicitly located the spatial sounds. We extracted subjectspecific mean response times and computed their across-subjects' mean separately for the 3 experiments (Psychophysics, fMRI, EEG), pre-and post-adaptation phases and 7 spatial locations (±12°, ±5°, ±2° and 0° visual angle). Supplementary Figure 2 shows the across subjects' mean (±SEM) responses across the 7 spatial locations for pre-and post-adaptation phases separately for the 3 experiments (Psychophysics, fMRI, EEG).

fMRI data acquisition and analysis fMRI multivariate decodingneurometric functions
Consistent with our behavioural analysis we assessed spatial encoding and recalibration in our ROIs using neurometric functions (NF 3 ). For this, we first binarized the decoded sound locations into 'decoded left' and 'decoded right' and computed the percentage of 'decoded right' for each sound location and participant. To enable reliable parameter estimates, we employed multi-condition fitting using the following constraints: (i) the JNDs (i.e., slope parameter) were constrained to be equal across all conditions; (ii) guess and lapse rates were set to be equal to each other and (iii) constrained to be equal across all conditions; (iv) lapse rates were set to be within 0 and 0.45 (to account for the lower signal-to-noise ratio of fMRI data); (v) the variance of the beta-distribution (i.e.,  in the beta-binomial model) was set equal across all conditions. Likelihood ratio test as described in the behavioural analysis section confirmed adequate goodness of fit for the neurometric functions in each ROI (i.e., p > 0.05 across all ROIs, Supplementary Table 3). We assessed whether AV-and VA-adaptation induced a shift in the decoded location of the unisensory auditory stimuli. Consistent with our behavioural analysis, we compared a 'static or no recalibration' model with a 'recalibration' model. The static model assumes no effect of audiovisual spatial adaptation on unisensory auditory spatial representations by constraining the point of subjective equality (PSE) for the pre-, postVA-, and postAV-adaptation PFs to be equal. The adaptation model includes three PSE values for the pre-, postVA-, and post-AVadaptation PFs thereby accommodating potential shifts in the PSE values as a result of audiovisual recalibration.

fMRI multivariate pattern analysisrepresentational dissimilarity analyses and multidimensional scaling
We characterized the spatial representations using representational similarity analysis (RSA, as implemented in the rsatoolbox 4 . For pre-adaptation, we computed a rank-transformed (and scaled between 0 and 1) 7 x 7 RDM based on the 7 auditory contrast images from the pre-adaptation phase. To characterize and visualize the effect of AV-and VA-adaptation on the representational geometry of spatial representations we computed a 14 x 14 matrix (i.e., 7 spatial locations for postVA-adaptation and 7 spatial locations for postAV-adaptation) where we rank-transformed (and scaled between 0 and 1) the 7 x 7 sub-components of the post-adaptation RDMs (i.e., separately for the postAV-and postVAadaptation). For visualization (see Figure 3C) we averaged the rank transformed matrices across participants.

Comparison of spatial hemifield and place code models
While accumulating research has shown that sound location is coded by broadly tuned neural populations in auditory cortices in mammals 5,6 , less is known about auditory spatial coding in parietal or frontal cortices from single cell neurophysiology. For instance, one previous neurophysiology study in non-human primates has suggested that neurons in parietal area VIP have narrow auditory receptive fields that are congruent with visual receptive fields 7 . Critically, the current study selected only nearcentral sound locations to optimize the characterization of recalibration rather than to dissociate between different spatial encoding principles. As we will show in the simulations below, the hemifield model and a place code model with narrow tuning functions make indistinguishable predictions for the pattern similarity structure over those near-central locations. We have therefore used the hemifield model as a generic perceptual spatial encoding model and note that comparable predictions could be made with the place code model.
In simulations, we investigated how the proportion of neurons tuned to different locations influences the pattern similarity structure. For the hemifield model we simulated 360 neurons with broad Gaussian tuning functions. The standard deviation was set to 64˚ as in our main study. The means of the tuning functions were sampled uniformly from 80˚ to 100 ˚ for the neuronal population tuned to the contralateral hemifield and from -80 ˚ to -100 ˚ azimuth for the neuronal population tuned to the ipsilateral hemifield. To illustrate how the ratio of ipsi-vs contralaterally tuned neurons affect the predictions of the model we performed two simulations with different ipsi/contra ratios: 30%/70% as suggested by previous research 8 and 50%/50%. For the spatial model based on the place code, we simulated 360 neurons with narrow Gaussian tuning functions (SD = 26˚) with means distributed between -179˚ and 180˚ azimuth. The means were distributed either uniformly in 1˚ increments or non-uniformly, with most of the neurons responding to the contralateral hemifield (see Supplementary Figure 1A).
As in our main analysis, for the pre-adaptation conditions we sampled neural responses from the seven sound locations (i.e., -12˚, -5˚, -2˚, 0˚, 2˚, 5˚, 12˚) in our paradigm. For the post-adaptation conditions, we sampled again from these seven locations for the non-recalibration model. For the recalibration model we sampled the neural responses from the above locations shifted by 2.3˚ to the right (postVAadapation) or left (postAV-adaptation). The shift by 2.3˚ was calculated as the difference between the across subjects' mean PSE values in postVA-and postAV-adaptation phases from the psychometric functions.
The hemifield models with ipsi/contra ratios of 30%/70% and 50%/50% (Supplementary Figure 3A, left) make different predictions for the regional mean BOLD-response (Supplementary Figure 3B), but not for the pattern similarity structure over locations (Supplementary Figure 3C). Likewise, the place code models with uniform and non-uniform sampling of neurons (Supplementary Figure 3A, right) with different tuning functions differ in their predictions for the regional mean BOLD-response (Supplementary Figure 3B), but not for the pattern similarity structure (Supplementary Figure 3C). In both cases, i.e., unequal ratios and non-uniform sampling can introduce an increase in the regional BOLD-response for sounds presented along the azimuth. By contrast, the similarity structure is relatively immune to these changes in model parameters.

Comparison of response time and decisional choice models
In addition to the spatial and decisional uncertainty models, we also developed two models based on observers' behavioural responses as dependent variables. The response time model encoded observers' response times (condition-specific mean response times averaged across participants) for each condition as predictors. The decisional choice model encoded the percentage left vs. right choices (averaged across participants) for each condition. For comparison, Supplementary Figure 4 shows the predictions of the spatial, the decisional uncertainty, the response time and the decisional choice model. We assessed the extent to which the predictions of the two behavioural models were distinct from our original spatial and decisional uncertainty models using Spearman correlation coefficient. Further, we assessed whether the decisional choice model explained additional variance in the regional mean fMRI or the fine grained fMRI and EEG activity patterns by comparing the winning model of each analysis from the main paper with the 'winning model extended with a decisional choice component'. The results are reported in Supplementary Note 2: Assessment of decisional choice model in fMRI and EEG and Supplementary Tables S6, S7, and S9.

Plotting of regional mean BOLD-responses (Figure 5A)
For visualization purposes only (i.e., Figure 5A), we averaged the BOLD-responses across hemispheres whilst controlling for the fact that the effects of spatial location and recalibration differ between the hemispheres as indicated by our model comparison results 9 Figure 5A).

Multivariate patternpattern component modelling of fMRI and EEG data
PCM estimates the likelihood of the observed neural activity under a multivariate linear model: is an (N x P) data matrix. N is equal to the number of conditions (i.e., 21) times the number of runs; P is equal to the number of measurement channels (fMRI voxels or EEG channels). Each row in includes the pre-whitened activity pattern, i.e., across-voxel parameter estimate patterns from the first-level GLM in fMRI and across-electrode evoked potentials in EEG. is a (N x 21 conditions) design matrix specifying the experimental conditions with 21 columns (i.e., 21 conditions = 7 sound locations x [pre, postVA, postAV]). is a matrix of true activity profiles (21 conditions x number of channels). is the design matrix for the patterns of no interest (e.g., condition-independent means of each voxelspecific activity profile). is the N x P matrix with the error terms. PCM treats the true activity profiles of each measurement channel (the columns of ) as random variables sampled from a multivariate normal distribution ~ ( 0, ) . Here, is the second moment matrix of activity profiles, which describes the similarity structure over the 21 conditions in the experiment.
can be expressed as a linear combination of second moment matrices (e.g., generated from the decisional and/or spatial model) with exp( h ) determining the weight of each second moment component to the second moment matrix estimated from the measured neural activity profiles over conditions. = ∑ exp ( h ) h ℎ 2 PCM estimates the marginal likelihood of the observed data given the model and the model parameters (i.e., the weights for the different second moment component matrices, signal strength, noise variance) by integrating over the actual activity profiles: Hence, the estimation of the individual activity profiles is not necessary and the second moment matrix (i.e., in our study a 21 x 21 matrix) is a sufficient statistic to compute the marginal likelihoods 10 .

Supplementary Tables
Supplementary Table 1 1.78 (±0.33) All behavioural results are reported as across-subjects' mean (± SEM). Number of subjects was 15 in psychophysics and 5 in fMRI and EEG experiments. The percentage of responses made when a response trial was indicated (Hit rate) and the corresponding sensitivity index (d') was computed separately for each experiment (Psychophysics, fMRI, and EEG) and experimental phases (preadaptation, adaptation and postAV/postVA-adaptation). In pre-and post-adaptation phases only, psychometric functions were fitted to the sound localization results. The point of subjective equality (PSE) values are reported based on the 'recalibration' model (see Methods). SEM: standard error of the mean.   Goodness of fit is assessed by comparing the loglikelihood ratio of the target relative to a saturated model for the original data relative to a bootstrapped null distribution of loglikelihood ratios.  9 Log e -Bayes factors (relative to the null model) for the linear mixed effects (LME) models fitted to the regional BOLD-response separately for HG, hA, IPS, IPL, FEF. (i) the spatial, decisional, and spatial + decisional models relative to a null model that includes only a constant. (ii) the spatial + decisional models that factorially manipulate whether recalibration is incorporated in the spatial and/or decisional components relative to the null model Anecdotal evidence: BF of 1 -3 (Log e BF of 0 -1.1); Moderate evidence: BF of 3 -10 (Log e BF of 1.1 -2.3); Strong evidence: BF of >10 (Log e BF of >2.3) 11 . S: spatial model without recalibration, D: decisional uncertainty model without recalibration, S+D: spatial model without recalibration + decisional uncertainty model without recalibration, S R +D: spatial model with recalibration + decisional uncertainty model without recalibration, S+D R : spatial model without recalibration + decisional uncertainty model with recalibration, S R +D R : spatial model with recalibration + decisional uncertainty model with recalibration null: null model, log e : natural logarithm, LME: linear mixed effects, ROI: region of interest, HG: Heschl's gyrus, hA: higher auditory cortex, IPS: intraparietal sulcus, IPL: inferior parietal lobule, FEF: frontal eye field. 3 Across-subjects' mean ± SEM of the relative Log e -Bayes factors for each pattern component model (relative to its particular baseline model as indicated in parentheses) fitted to the fMRI activation patterns separately for HG, hA, IPS, IPL, FEF. (i) the spatial, decisional, and spatial + decisional PCMs relative to a null model that assumes activity patterns over all sound locations are independent. (ii) the spatial + decisional PCMs that factorially manipulate whether recalibration is incorporated in the spatial and/or decisional components relative to the spatial + decisional uncertainty model without recalibration. Anecdotal evidence: BF of 1 -3 (Log e BF of 0 -1.1); Moderate evidence: BF of 3 -10 (Log e BF of 1.1 -2.3); Strong evidence: BF of >10 (Log e BF of >2.3) 11 . S: spatial model without recalibration, D: decisional uncertainty model without recalibration, S+D: spatial model without recalibration + decisional uncertainty model without recalibration, S R +D: spatial model with recalibration + decisional uncertainty model without recalibration, S+D R : spatial model without recalibration + decisional uncertainty model with recalibration, S R +D R : spatial model with recalibration + decisional uncertainty model with recalibration, Free: free model, null: null model, rel: relative to, log e : natural logarithm, PCM: pattern component modelling, ROI: region of interest, HG: Heschl's gyrus, hA: higher auditory cortex, IPS: intraparietal sulcus, IPL: inferior parietal lobule, FEF: frontal eye field. Log e -Bayes factors (relative to the null model that includes only a constant) for the linear mixed effects (LME) models fitted to the regional BOLD-response separately for HG, hA, IPS, IPL, and FEF. (i) the best performing model and its extension with decisional choice model from the space encoding analysis (Supplementary Table 4). (ii) the best performing model and its extension with decisional choice model with recalibration from the recalibration analysis (Supplementary Table 4  Across-subjects' mean ± SEM of the relative Log e -Bayes factors for each pattern component model (relative to its particular baseline model as indicated in parentheses) fitted to the EEG activity patterns separately for four time windows. (i) the spatial, decisional, and spatial + decisional PCMs relative to a null model that assumes activity patterns over all sound locations are independent. (ii) the spatial + decisional PCMs that factorially manipulate whether recalibration is incorporated in the spatial and/or decisional components relative to the spatial + decisional uncertainty model without recalibration. (iii) the PCMs including pattern components generated from the BOLD-response patterns of one of the four ROIs (combined over both hemispheres) relative to the HG model: hA, IPS, IPL and FEF Anecdotal evidence: BF of 1 -3 (Log e BF of 0 -1.1); Moderate evidence: BF of 3 -10 (Log e BF of 1.1 -2.3); Strong evidence: BF of >10 (Log e BF of >2.3) 11 . S: spatial model without recalibration, D: decisional uncertainty model without recalibration, S+D: spatial model without recalibration + decisional uncertainty model without recalibration, S R +D: spatial model with recalibration + decisional uncertainty model without recalibration, S+D R : spatial model without recalibration + decisional uncertainty model with recalibration, S R +D R : spatial model with recalibration + decisional uncertainty model with recalibration, Free: free model, null: null model, rel: relative to, log e : natural logarithm, PCM: pattern component modelling, ROI: region of interest, HG: Heschl's gyrus, hA: higher auditory cortex, IPS: intraparietal sulcus, IPL: inferior parietal lobule, FEF: frontal eye field.

Supplementary Figures
Supplementary Figure 1: The influence of audiovisual integration and recalibration on perceived sound location, psychometric function and decisional uncertainty. (A) In the pre-adaptation phase, stimuli emanating from the centre (0° visual angle, see top) should be perceived equally often as 'right' or 'left' (50% vs. 50%) as illustrated by the psychometric function (PF, see middle). This point of subjective equality (PSE) is also associated with the greatest decisional uncertainty as illustrated by the predictions of the decisional uncertainty model (see bottom). (B) In the adaptation phase the observer is presented with auditory and visual signals in synchrony. Critically, the visual stimuli are presented with a displacement towards the left (i.e. VA-adaptation, here shown in B) or the right of the sound location. As a result of audiovisual integration, observers perceive the sound shifted towards the visual stimulus (i.e. here: shifted towards the left), a phenomenon referred to as ventriloquist effect. (C) As a result of recalibration, this shift in perceived sound location towards the visual stimulus is also preserved in the postVA or postAV-adaptation phase, when the sound is presented alone (i.e. ventriloquist aftereffect). For instance, in the postVA-adaptation phase the perceived sound location is shifted towards the left (i.e. the location of the previously presented visual stimulus), such that a true sound location of +2.5° visual angle is now perceived in the centre (i.e. perceived equally often as 'right' or 'left'). Therefore, both the psychometric function, particularly its PSE, and the associated peak of decisional uncertainty is shifted towards the right after recalibration towards the left.
Supplementary Figure 2. Group response times in psychophysics, fMRI, and EEG experiments. Acrosssubjects' mean reaction times (±SEM) as a function of the 7 spatial locations (±12°, ±5°, ±2° and 0°) along the azimuth separately for pre-, postAV-, and postVA-adaptation phases and experiments (Psychophysics, fMRI, and EEG). We subtracted the across-conditions mean response time for each participant from their condition-specific mean response times prior to averaging response times across participants. As a result, response times are positive close to the decision boundary, but negative far away from the decision boundary. As expected, the maximum of the response time curves that is thought to reflect decisional uncertainty shifts depending on the direction of audiovisual recalibration. It is shifted towards the left after postAV-adaptation and towards the right after postAV-adaptation. Source data are provided as a Source Data file.
biased towards the contralateral hemifield. (C) Predicted representational dissimilarity matrices (RDM) based on the individual model neural activity profiles across spatial locations (-12° to 12°) and experimental phases (pre-, postVA-, and postAV-adaptation). We simulated RDMs from the spatial hemifield (left) and the place code (right) model for (i) top row: no recalibration, i.e., without a representational shift and (ii) bottom row: with recalibration, i.e., with a representational shift. Both models make nearly indistinguishable predictions for the RDMs regardless of exact parameter settings (e.g., uniform vs. non uniform sampling). Solid white lines delineate the sub-RDM matrices that show the representational dissimilarities for different stimulus locations within and between different experimental phases. Diagonal dashed white lines highlight the RDM dissimilarity values for two identical physical locations of the postAV-and the postVA-adaptation phases. Comparing the RDMs with and without recalibration along those dashed white lines shows how the shift in spatial representations towards the previously presented visual stimulus alters the representational dissimilarity of corresponding stimulus locations in postAV-and postVA-adaptation phases, while the off-diagonals show the dissimilarity values for neighbouring spatial locations. Source data are provided as a Source Data file.
Supplementary Figure 4. Comparison of spatial, decisional uncertainty, decisional choice and reaction time models.
(A) Neural models: The hemifield model encodes spatial location in the relative activity of two subpopulations of neurons each broadly tuned either to the ipsi-or contra-lateral hemifield. The ratio of the ipsi-and contralaterally tuned neurons was set to 30%/70% (dark grey) and 50%/50% (light grey). The place code model encodes spatial location in the activity of a large population of neurons each narrowly tuned to a particular spatial location. We sampled the means of these tuning functions across neurons either uniformly (light grey) or non-uniformly, i.e., with a contralateral bias (dark grey) over space. Note we did not plot neural models underlying decisional choice and reaction time models as those are based on the group average psychometric functions and reaction times, respectiely. (B) Predicted mean BOLD-response as a function of sound location along the azimuth in a left hemisphere region for pre-, postVA-and postAV-adaptation. Both the spatial hemifield model and the place code model predicts a BOLD-response that increases across spatial locations only when the ratio across the two populations (hemifield) or the means of the narrow tuning functions (place code) are biased towards the contralateral hemifield. (C) Predicted representational dissimilarity matrices (RDM) based on the individual model neural activity profiles across spatial locations (-12° to 12°) and experimental phases (pre-, postVA-, and postAV-adaptation). We simulated RDMs from the spatial hemifield (left) and the place code (right) model for (i) top row: no recalibration, i.e., without a representational shift and (ii) bottom row: with recalibration, i.e., with a representational shift. Both models make nearly indistinguishable predictions for the RDMs regardless of exact parameter settings (e.g., uniform vs. non uniform sampling). Solid white lines delineate the sub-RDM matrices that show the representational dissimilarities for different stimulus locations within and between different experimental phases. Diagonal dashed white lines highlight the RDM dissimilarity values for two identical physical locations of the postAV-and the postVA-adaptation phases. Comparing the RDMs with and without recalibration along those dashed white lines shows how the shift in spatial representations towards the previously presented visual stimulus alters the representational dissimilarity of corresponding stimulus locations in postAV-and postVA-adaptation phases, while the off-diagonals show the dissimilarity values for neighbouring spatial locations. Source data are provided as a Source Data file.